CLI Flags Now Working

All three speculative draft model offload flags are now fully functional:

• --override-tensor-draft: Specify tensor buffer type overrides for the draft model
• --cpu-moe-draft: Keep all MoE weights in CPU for the draft model
• --n-cpu-moe-draft N: Keep MoE weights of first N layers in CPU for the draft model

Example Usage

# Use different tensor overrides for draft model
./llama-speculative --override-tensor-draft "*.weight=CPU" --model-draft draft.gguf --model main.gguf

# Keep MoE weights in CPU for draft model
./llama-server --cpu-moe-draft --model-draft draft.gguf --model main.gguf

# Combined usage
./llama-speculative --override-tensor-draft "blk.0.*=CPU" --n-cpu-moe-draft 3 --model-draft draft.gguf --model main.gguf

Entrypoints Updated

All speculative decoding entrypoints properly apply draft-specific tensor overrides:

• examples/speculative/speculative.cpp
• examples/speculative-simple/speculative-simple.cpp
• tools/server/server.cpp

Validation Results

• All executables build successfully
• CLI help shows all flags correctly
• Flag validation works (shows available buffer types on error)
• No assertion failures when using draft tensor overrides
• Draft overrides are isolated from main model overrides
• All flags can be used together without conflicts

The implementation ensures draft model tensor overrides are completely independent from main model overrides, enabling flexible heterogeneous hardware setups and advanced MoE configurations for speculative decoding workflows.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.